Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

do NOT review - offloading for other recipes #1578

Closed
wants to merge 10 commits into from
Closed

do NOT review - offloading for other recipes #1578

wants to merge 10 commits into from

Conversation

felipemello1
Copy link
Contributor

@felipemello1 felipemello1 commented Sep 13, 2024 •
edited
Loading

Context

What is the purpose of this PR? Is it to

  • add a new feature
  • fix a bug
  • update tests and/or documentation
  • other (please add here)

Please link to any issues this PR addresses.

Changelog

What are the changes made in this PR?

Test plan

tune run full_finetune_single_device --config llama3_1/8B_full_single_device \
dataset._component_=torchtune.datasets.text_completion_dataset \
dataset.source=text \
dataset.split="train" \
dataset.data_files="/tmp/fake_long_dataset/train/fake_long_dataset.txt" \
tokenizer.max_seq_len=8192 \
batch_size=4 \
max_steps_per_epoch=10 \
metric_logger=torchtune.training.metric_logging.WandBLogger \
gradient_accumulation_steps=1 \
epochs=1 \
log_peak_memory_stats=True \
optimizer=bitsandbytes.optim.PagedAdamW8bit \
optimizer_in_bwd=True \
compile=True \
enable_activation_checkpointing=True \
enable_activation_offloading=True \
offload_with_streams=True
image image

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Sep 13, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1578

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7aa7e49 with merge base 4fbe7b2 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 13, 2024
@codecov-commenter
Copy link

codecov-commenter commented Sep 13, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 4.97238% with 172 lines in your changes missing coverage. Please review.

Project coverage is 72.29%. Comparing base (ee343e6) to head (7aa7e49).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
torchtune/training/_activation_offloading.py 5.88% 128 Missing ⚠️
recipes/full_finetune_distributed.py 0.00% 11 Missing ⚠️
recipes/full_finetune_single_device.py 0.00% 11 Missing ⚠️
recipes/lora_finetune_distributed.py 0.00% 11 Missing ⚠️
recipes/lora_finetune_single_device.py 0.00% 11 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1578      +/-   ##
==========================================
- Coverage   73.08%   72.29%   -0.79%     
==========================================
  Files         289      290       +1     
  Lines       14157    14348     +191     
==========================================
+ Hits        10346    10373      +27     
- Misses       3811     3975     +164

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@joecummings joecummings mentioned this pull request Oct 15, 2024
Closed
34 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@felipemello1 @codecov-commenter @facebook-github-bot @janeyx99